Generally speaking a global computer network, e.g., the Internet, is formed of a plurality of computers coupled to a communication line for communicating with each other. Each computer is referred to as a network node. Some nodes serve as information bearing sites while other nodes provide connectivity between end users and the information bearing sites.
The explosive growth of the Internet makes it an essential component of every business, organization and institution strategy, and leads to massive amounts of information being placed in the public domain for people to read and explore. The type of information available ranges from information about companies and their products, services, activities, people and partners, to information about conferences, seminars, and exhibitions, to news sites, to information about universities, schools, colleges, museums and hospitals, to information about government organizations, their purpose, activities and people. The Internet has become the venue of choice for every organization for providing pertinent, detailed and timely information about themselves, their cause, services and activities.
The Internet essentially is the network infrastructure that connects geographically dispersed computer systems. Every such computer system may contain publicly available (shareable) data that are available to users connected to this network. However, until the early 1990""s there was no uniform way or standard conventions for accessing this data. The users had to use a variety of techniques to connect to remote computers (e.g. telnet, ftp, etc) using passwords that were usually site-specific, and they had to know the exact directory and file name that contained the information they were looking for.
The World Wide Web (WWW or simply Web) was created in an effort to simplify and facilitate access to publicly available information from computer systems connected to the Internet. A set of conventions and standards were developed that enabled users to access every Web site (computer system connected to the Web) in the same uniform way, without the need to use special passwords or techniques. In addition, Web browsers became available that let users navigate easily through Web sites by simply clicking hyperlinks (words or sentences connected to some Web resource).
Today the Web contains more than one billion pages that are interconnected with each other and reside in computers all over the world (thus the term xe2x80x9cWorld Wide Webxe2x80x9d). The sheer size and explosive growth of the Web has created the need for tools and methods that can automatically search, index, access, extract and recombine information and knowledge that is publicly available from Web resources.
As used herein, the following terms have the indicated definitions.
Web Domain
Web domain is an Internet address that provides connection to a Web server (a computer system connected to the Internet that allows remote access to some of its contents).
URL
URL stands for Uniform Resource Locator. Generally, URLs have three parts: the first part describes the protocol used to access the content pointed to by the URL, the second contains the domain directory in which the content is located, and the third contains the file that stores the content:
 less than protocol greater than :  less than domain greater than   less than directory greater than   less than file greater than 
For example:
www.corex.com/bios.html
www.cardscan.com/index.html
fn.cnn.com/archives/may99/pr37.html
shiva.lin.com/soft/words.zip
Commonly, the  less than protocol greater than  part may be missing. In that case, modern Web browsers access the URL as if the http:// prefix was used. In addition, the  less than file greater than  part may be missing. In that case, the convention calls for the file xe2x80x9cindex.htmlxe2x80x9d to be fetched.
For example, the following are legal variations of the previous example URLs:
www.corex.com/bios.html
www.cardscan.com
fn.cnn.com/archives/may99/pr37.html
ftp://shiva.lin.com/soft/words.zip
Web Page
Web page is the content associated with a URL. In its simplest form, this content is static text, which is stored into a text file indicated by the URL. However, very often the content contains multi-media elements (e.g. images, audio, video, etc) as well as non-static text or other elements (e.g. news tickers, frames, scripts, streaming graphics, etc). Very often, more than one file forms a Web page, however, there is only one file that is associated with the URL and which initiates or guides the Web page generation.
Web Browser
Web browser is a software program that allows users to access the content stored in Web sites. Modem Web browsers can also create content xe2x80x9con the flyxe2x80x9d, according to instructions received from a Web site. This concept is commonly referred to as xe2x80x9cdynamic page generationxe2x80x9d. In addition, browsers can commonly send information back to the Web site, thus enabling two-way communication of the user and the Web site.
The concept of Web site is closely linked to the concept of Web domain, however they have some important differences. Web site is a more abstract term, which refers to a collection of Web pages usually put together, owned, and maintained by a single entity (which can be company, organization, institution, person, etc). Very often, a Web site resides on a particular Web domain, however there are exceptions when the content of a site is distributed among several domains. In general, every Web site has at least one domain, however the opposite is not always true (there are domains with no corresponding Web site).
In many cases a Web site can be viewed as a book where each chapter provides information about a different but related topic and where it is assumed that the reader will correlate information from one page to the other so that there is no need to repeat the same information on every page. For example, a company Web site may contain pages dedicated to the company""s mission, history, products, services, management, contact information, news, etc. The company name might not even be mentioned when talking about products or its management and is assumed to be conveyed by the fact that the present/currently displayed page is part of the company Web site.
The content published in a Web site can vary considerably in both its type and format: it may be news, information, fiction, statements, etc. given as text, pictures, video, audio, graphics, etc. The Web site content owner is the legal entity (individual or institution, company, organization, etc) which publishes and is legally responsible for this content.
Identifying the content owner allows tagging the information with its source, in order to filter it, estimate its value and accuracy, or use it according to attributes associated with this source. In addition to that, very often information collected from a Web site is incomplete if the site owner is unknown. For example, in Web mining projects that automatically collect people or Organization information (see Inventions 5, 6 and 7 as disclosed in the related Provisional Application No. 60/221,750 filed on Jul. 31, 2000 for a xe2x80x9cComputer Database Method and Apparatusxe2x80x9d) it is essential to know who is the legal entity that publishes this information. The reason is that very often, the information is given without explicitly specifying the entity that this information is about; for example, Corex Corporation publishes in its Web site a list of directors, under the simple header xe2x80x9cBoard of Directorsxe2x80x9d, without explicitly specifying xe2x80x9cBoard of Directors of Corex Corporationxe2x80x9d. Human readers can deduce the implicit information, however, it is very difficult for automated computer programs to connect the information to the correct Organization name.
In addition to the above, identifying the owner of a Web site has value in itself, in composing a list of {Web domain, legal owner} pairs. There are already commercial companies that sell lists of this kind (e.g. Network Solutions Inc), however, they either create them manually, limit the list to only part of all existing Web domains, or do not update them on a regular basis to reflect frequent changes.
The main purpose of the present invention is to automatically identify the content owner of a Web site, given a set of Web pages from that site. Statistical algorithms are used to make inferences based on an array of tests and candidate strings that are extracted from the given pages. The whole process is completely automated, so that it can run without human supervision or labor. In this way, the present invention method coupled with a Web robot (see invention 4 as disclosed in the related Provisional Application No. 60/221,750) is capable of producing the complete list of {Web domain, legal owner} for all existing domains in the Web, in a totally automated fashion.
In a preferred embodiment, a method of identifying content owner of a Web site comprises the computer-implemented steps of: (i) collecting candidate names from a subject Web site; (ii) for each candidate name, running tests having test results which enable quantitative evaluation of the candidate name being content owner name of the subject Web site; and (iii) mathematically combining the test results into an indication of content owner name.
In particular, the step of running tests includes employing tests that enable statistical evaluation of each candidate name being the content owner name. Preferably, the step of combining the test results includes using a Bayesian network. In a further step, the Bayesian network is trained using a training set of Web sites with respective known content owner names such that statistics on the test results are collected on the training set of Web sites.
The invention method may further comprise the step of providing a confidence level of the indicated content owner name.
In the preferred embodiment, the step of collecting candidate names includes using software robots, Web crawlers, Web spiders and/or agents. Further, the step of collecting candidate names includes collecting noun phrases that are capitalized and in particular have a generic term that is indicative of an organization. For example, the terms (or the abbreviations thereof) xe2x80x9cagencyxe2x80x9d, xe2x80x9cassociationxe2x80x9d, xe2x80x9cbankxe2x80x9d, xe2x80x9cclubxe2x80x9d, xe2x80x9ccompanyxe2x80x9d, xe2x80x9ccorporationxe2x80x9d, xe2x80x9ccouncilxe2x80x9d, xe2x80x9centerprisexe2x80x9d, xe2x80x9cfoundationxe2x80x9d, xe2x80x9chospitalxe2x80x9d, xe2x80x9cincorporatedxe2x80x9d, xe2x80x9clabsxe2x80x9d, xe2x80x9climitedxe2x80x9d, xe2x80x9cpartnersxe2x80x9d, xe2x80x9cventuresxe2x80x9d and the like are commonly used to indicate an organization. The invention collects noun phrases that include such terms and uses the collected noun phrases as candidate names.
In addition, the step of combining the test results to identify the content owner name includes determining when no said content owner name exists at the subject Web site. In that case, the step of running tests enables statistical evaluation of the candidate name being the content owner name; and the step of determining no content owner name exists at the subject Web site is based on no candidate name having a probability within a predetermined threshold. Preferably, the threshold is statistically predefined by a desired ratio of false positives to false negatives.
In accordance with the present invention, computer apparatus for identifying the content owner of a Web site comprises: means for collecting candidate names from a subject Web site; and a test module responsive to the means for collecting and including a plurality of processor-executed tests having test results which enable quantitative (e.g., statistical) evaluation of the candidate name being content owner name of the subject Web site. For each candidate name, the test module runs the tests and combines the test results into an indication of content owner name or an indication that no content owner name exists at the subject Web site.
The test module preferably employs a Bayesian network to mathematically combine the test results. A training member trains the Bayesian network using a training set of Web sites with known content owner names. The training module collects statistics on the test results from the training set of Web sites. The test module further provides a confidence level of the indicated content owner name.
The means for collecting candidate names includes software robots, Web crawlers, Web spiders and/or agents.
The method and apparatus includes storing determined indications of content owner names in a manner and means that enables cross referencing with indications of respective Web sites. To that end, a database (or similar storage means) containing indications of content owner name correlated with a respective Web site is formed.